Deep learning network to correct axial and coronal eye motion in 3D OCT retinal imaging

Optical Coherence Tomography (OCT) is one of the most important retinal imaging technique. However, involuntary motion artifacts still pose a major challenge in OCT imaging that compromises the quality of downstream analysis, such as retinal layer segmentation and OCT Angiography. We propose deep learning based neural networks to correct axial and coronal motion artifacts in OCT based on a single volumetric scan. The proposed method consists of two fully-convolutional neural networks that predict Z and X dimensional displacement maps sequentially in two stages. The experimental result shows that the proposed method can effectively correct motion artifacts and achieve smaller error than other methods. Specifically, the method can recover the overall curvature of the retina, and can be generalized well to various diseases and resolutions

Optimization and learning based video coding

The complexity of video coding standards has increased significantly from H.262/MPEG-2 to H.264/AVC in order to increase coding efficiency. Complexity mainly was increased more by architecture than by algorithms: One 16x16 MB type in MPEG-2 was partitioned into various MB types such as 16x16, 8x16, 8x8, 4x4. Half pixel accuracy motion estimation was extended to support quarter pixel accuracy, and various simple directional filters were applied for intra prediction. In this dissertation, we consider optimization and learning methods to solve video coding problems. In our approaches, complexity is mainly increased by algorithms to improve coding efficiency. Especially, we apply these methods for the Rate-Distortion (RD) optimization problem in H.264 and intra prediction as a new video coding scheme because they are highly related with numerical optimization and regression theories. For the RD optimization problem, we propose a general framework with consideration of temporal prediction dependency using the primal-dual decomposition and subgradient projection methods. As a result, optimality conditions among the Lagrange multipliers $\lambda$ are derived for the optimal bit allocation. The proposed method is compared with the Rate Control (RC) algorithm in the reference software model (JM model) of H.264. In order to reduce the complexity of the proposed method, an adaptive Lagrange multiplier selection method is proposed in the RC algorithm using the Classification-Maximization (CM) algorithm. In addition, two variations of the CM algorithm, that is, Relaxed CM (RCM) and Incremental CM (ICM) are proposed to improve the performance and avoid iterations. We compare [lambda] of the proposed adaptive Lagrange multiplier selection methods with ones of the JM model and the greedy search. Finally, we propose a new video coding scheme using learning methods. In particular, learning methods such as support vector regression and locally weighted learning are applied for intra prediction by means of batch and online learning. We present that online learning based intra prediction is better for video coding because of limited training time and nonstationary video sequences even though batch learning based intra prediction can achieve significant improvement in low- motion sequences. Experimental results show that online learning based video coding is promising for future video codin

Optimization and learning based video coding

The complexity of video coding standards has increased significantly from H.262/MPEG-2 to H.264/AVC in order to increase coding efficiency. Complexity mainly was increased more by architecture than by algorithms: One 16x16 MB type in MPEG-2 was partitioned into various MB types such as 16x16, 8x16, 8x8, 4x4. Half pixel accuracy motion estimation was extended to support quarter pixel accuracy, and various simple directional filters were applied for intra prediction. In this dissertation, we consider optimization and learning methods to solve video coding problems. In our approaches, complexity is mainly increased by algorithms to improve coding efficiency. Especially, we apply these methods for the Rate-Distortion (RD) optimization problem in H.264 and intra prediction as a new video coding scheme because they are highly related with numerical optimization and regression theories. For the RD optimization problem, we propose a general framework with consideration of temporal prediction dependency using the primal-dual decomposition and subgradient projection methods. As a result, optimality conditions among the Lagrange multipliers $\lambda$ are derived for the optimal bit allocation. The proposed method is compared with the Rate Control (RC) algorithm in the reference software model (JM model) of H.264. In order to reduce the complexity of the proposed method, an adaptive Lagrange multiplier selection method is proposed in the RC algorithm using the Classification-Maximization (CM) algorithm. In addition, two variations of the CM algorithm, that is, Relaxed CM (RCM) and Incremental CM (ICM) are proposed to improve the performance and avoid iterations. We compare [lambda] of the proposed adaptive Lagrange multiplier selection methods with ones of the JM model and the greedy search. Finally, we propose a new video coding scheme using learning methods. In particular, learning methods such as support vector regression and locally weighted learning are applied for intra prediction by means of batch and online learning. We present that online learning based intra prediction is better for video coding because of limited training time and nonstationary video sequences even though batch learning based intra prediction can achieve significant improvement in low- motion sequences. Experimental results show that online learning based video coding is promising for future video codin

A Convolutional Neural Network Pipeline For Multi-Temporal Retinal Image Registration

A sequence of images is usually captured to observe the change of health status in medical diagnosis. However, an image sequence taken over year usually suffers from severe deformation, making it time-consuming for physicians to match corresponding patterns. In this paper, we propose a coarse-to-fine pipeline for retinal image registration based on convolutional neural network. By leveraging the three components of the pipeline: feature matching, outlier rejection, and local registration, we recover the deformation and accurately align multi-temporal image sequences. Experimental results show that the proposed network is robust to severe deformation as well as illumination and contrast variations. With the proposed registration pipeline, the change of image patterns over time can be identified through visual analysis

A high-throughput technique to map cell images to cell positions using a 3D imaging flow cytometer.

We develop a high-throughput technique to relate positions of individual cells to their three-dimensional (3D) imaging features with single-cell resolution. The technique is particularly suitable for nonadherent cells where existing spatial biology methodologies relating cell properties to their positions in a solid tissue do not apply. Our design consists of two parts, as follows: recording 3D cell images at high throughput (500 to 1,000 cells/s) using a custom 3D imaging flow cytometer (3D-IFC) and dispensing cells in a first-in-first-out (FIFO) manner using a robotic cell placement platform (CPP). To prevent errors due to violations of the FIFO principle, we invented a method that uses marker beads and DNA sequencing software to detect errors. Experiments with human cancer cell lines demonstrate the feasibility of mapping 3D side scattering and fluorescent images, as well as two-dimensional (2D) transmission images of cells to their locations on the membrane filter for around 100,000 cells in less than 10 min. While the current work uses our specially designed 3D imaging flow cytometer to produce 3D cell images, our methodology can support other imaging modalities. The technology and method form a bridge between single-cell image analysis and single-cell molecular analysis

Artificial Intelligence for Automated Overlay of Fundus Camera and Scanning Laser Ophthalmoscope Images.

PurposeThe purpose of this study was to evaluate the ability to align two types of retinal images taken on different platforms; color fundus (CF) photographs and infrared scanning laser ophthalmoscope (IR SLO) images using mathematical warping and artificial intelligence (AI).MethodsWe collected 109 matched pairs of CF and IR SLO images. An AI algorithm utilizing two separate networks was developed. A style transfer network (STN) was used to segment vessel structures. A registration network was used to align the segmented images to each. Neither network used a ground truth dataset. A conventional image warping algorithm was used as a control. Software displayed image pairs as a 5 × 5 checkerboard grid composed of alternating subimages. This technique permitted vessel alignment determination by human observers and 5 masked graders evaluated alignment by the AI and conventional warping in 25 fields for each image.ResultsOur new AI method was superior to conventional warping at generating vessel alignment as judged by masked human graders (P < 0.0001). The average number of good/excellent matches increased from 90.5% to 94.4% with AI method.ConclusionsAI permitted a more accurate overlay of CF and IR SLO images than conventional mathematical warping. This is a first step toward developing an AI that could allow overlay of all types of fundus images by utilizing vascular landmarks.Translational relevanceThe ability to align and overlay imaging data from multiple instruments and manufacturers will permit better analysis of this complex data helping understand disease and predict treatment